Fix GCS IO manager auth + carry stranded GeoPackage fixes to main#77
Merged
Conversation
For GeoPackage, PUT file.gpkg?configure=all created the datastore but did not publish a layer, leaving a store with no associated layer. Switch to configure=none and POST the featuretype explicitly (name/nativeName = gpkg table name, srs EPSG:4326), folding title/abstract into that call. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sites with an elevation produce 3D point geometry ([lon, lat, elev]). GeoServer's GeoPackage reader then fails computing bounds: MismatchedDimensionException: Argument "WGS 84" has 3 dimensions, while 2 was expected. Call force_2d() on the GeoDataFrame before to_file(GPKG). Elevation is still carried as an attribute; only the geometry dimensionality changes. The GeoJSON product on GCS is unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The GCSPickleIOManager used dagster_gcp's GCSResource, whose get_client() goes through Application Default Credentials. Dagster+ serverless has no ADC, so asset I/O failed: google.auth.exceptions.DefaultCredentialsError: Your default credentials were not found. Add AuthedGCSResource (dagster_gcp GCSResource subclass) that builds the client from GCP_SERVICE_ACCOUNT_KEY, the same secret the product-upload GCSResource already uses, and wire it into the IO manager. Factored the client builder into _storage_client() shared by both. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Your pull request is automatically being deployed to Dagster Cloud.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three fixes, all needed on
main.1. GCS IO manager auth (the current failure)
The
GCSPickleIOManager(PR #76) useddagster_gcp'sGCSResource, whoseget_client()uses Application Default Credentials. Dagster+ serverless has no ADC. AddedAuthedGCSResource— adagster_gcpGCSResourcesubclass whoseget_client()builds the client fromGCP_SERVICE_ACCOUNT_KEY, the same secret the product-uploadGCSResourcealready uses. Factored the client builder into a shared_storage_client().2 + 3. GeoPackage fixes that never reached main
PR #75 merged only its first commit; these two follow-ups were left on the post-merge branch:
configure=allcreated the gpkg datastore but no layer. Nowconfigure=none+ explicitPOST .../featuretypes.MismatchedDimensionException: "WGS 84" has 3 dimensions.force_2d()before writing the gpkg.Verified: definitions load (38 assets), io_manager.gcs is AuthedGCSResource, flake8 clean.
🤖 Generated with Claude Code